Image Enhancement Devices With Gaze Tracking

ABSTRACT

An electronic device may have a display and a camera. Control circuitry in the device can gather information on a user&#39;s point of gaze using a gaze tracking system and other sensors, can gather information on the real-world image such as information on content, motion, and other image attributes by analyzing the real-world image, can gather user vision information such as user acuity, contrast sensitivity, field of view, and geometrical distortions, can gather user input such as user preferences and user mode selection commands, and can gather other input. Based on the point-of-gaze information and/or other gathered information, the control circuitry can display the real-world image and supplemental information on the display. The supplemental information can include augmentations such as icons, text labels, and other computer-generated text and graphics overlaid on the real world image and can include enhanced image content such as magnified portions of the real-world image.

This application claims the benefit of provisional patent applicationNo. 62/643,023, filed Mar. 14, 2018, which is hereby incorporated byreference herein in its entirety.

BACKGROUND

This relates generally to electronic devices and, more particularly, toelectronic devices that display mixed reality content.

Mixed reality devices may have cameras and displays. The cameras cancapture images of the real world. Content such as text and graphics canthen be overlaid on the real-world images to produce mixed-realityimages.

It can be challenging to present mixed reality content to a user. Thepresentation of mixed-reality images to a user may, for example, bedisrupted by head and eye movement. Users may have different visualcapabilities. If care is not taken, mixed-reality images will bepresented that cause disorientation or motion sickness and that make itdifficult for a user to identify items of interest.

SUMMARY

An electronic device may have a display and a camera. The camera may beused to capture a real-world image. The real-world image can bedisplayed on the display for a user.

Control circuitry in the device can gather information on the user'spoint of gaze using a gaze tracking system and other sensors. Thecontrol circuitry can also gather information on the real-world imagesuch as information on content, motion, and other image attributes byanalyzing the real-world image. User vision information such as uservisual acuity, contrast sensitivity, field of view, and geometricaldistortions can be provided to the control circuitry. Input-outputcircuitry can be used by the control circuitry to gather user input suchas user preferences and user mode selection commands and can be used togather location information and other input

Based on the point-of-gaze information and/or other gatheredinformation, the control circuitry can display supplemental informationon the real-world image on the display. The supplemental information caninclude augmentations such as icons, text labels, and othercomputer-generated text and graphics overlaid on the real world imageand can include enhanced image content such as magnified portions of thereal-world image. If desired, the real-world image may be globally orlocally enhanced by adjusting contrast, altering image color, enhancingedges, adjusting for geometrical vision distortion, and performing otherimage enhancements on the real-world image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of an illustrative electronic device inaccordance with an embodiment.

FIG. 2 is a diagram of gaze tracking circuitry in accordance with anembodiment.

FIG. 3 is a diagram of an illustrative real-world image and supplementalinformation displayed on the real-world image in accordance with anembodiment.

FIGS. 4, 5, and 6 are diagrams showing illustrative image enhancementzones on a display in accordance with an embodiment.

FIG. 7 is a flow chart of illustrative operations involved in providinga user with mixed reality content in accordance with an embodiment.

DETAILED DESCRIPTION

Electronic devices may be provided with displays. The displays may beused in displaying supplemental content overlaid on real-world content.The supplemental content may, for example, include augmentation such ascomputer-generated text and graphics and may include enhanced imagecontent such as real-world image data that has been modified byadjusting image attributes such as contrast, magnification, and otherproperties.

An electronic device may use information such as information from a gazetracking system and other information in displaying image enhancements.As an example, a portion of a real-world image may be magnified for alow vision user only when a user's point of gaze is relatively static toavoid inducing motion sickness effects. An electronic device may analyzea real-world image to identify content of interest (e.g., the locationand motion of objects, the identity of faces and other items, thepresence of text, the location of the edges of an object, etc.). Ifdesired, information such as a user's eyeglass prescription and otheruser vision data may be used in displaying image enhancements. Forexample, text in an image can be magnified if a user is visuallyimpaired (e.g., if a user has low spatial acuity) or image content canbe distorted to compensate for geometric distortion created by theuser's vision. Image enhancements may be applied globally to areal-world image or may be displayed in appropriate image enhancementregions. If desired, user input and other information can also be usedin displaying supplemental information on a real-world image.

A schematic diagram of an illustrative electronic device of the typethat may be used in presenting a user with mixed reality content isshown in FIG. 1. Electronic device 10 may be a head-mounted device(head-mounted display), a cellular telephone, a tablet computer, ahead-up display (e.g., a mixed reality display in an automobile or othervehicle), a laptop or desktop computer, a television, a wrist watch, orother electronic equipment. As shown in FIG. 1, electronic device 10 mayhave control circuitry 20. Control circuitry 20 may include storage andprocessing circuitry for controlling the operation of device 10.Circuitry 20 may include storage such as hard disk drive storage,nonvolatile memory (e.g., electrically-programmable-read-only memoryconfigured to form a solid state drive), volatile memory (e.g., staticor dynamic random-access-memory), etc. Processing circuitry in controlcircuitry 20 may be based on one or more microprocessors,microcontrollers, digital signal processors, baseband processors, powermanagement units, audio chips, graphics processing units, applicationspecific integrated circuits, and other integrated circuits. Softwarecode may be stored on storage in circuitry 20 and run on processingcircuitry in circuitry 20 to implement control operations for device 10(e.g., operations associated with capturing real-world images with acamera, data gathering operations, operations involved in monitoring auser's point of gaze, operations involved in generating content overlaidon top of real-world images, operations associated with performingpattern recognition and other image processing operations on real-worldimages, operations associated with displaying images with real-worldcontent and overlaid content, etc.)

Device 10 may include input-output circuitry 22. Input-output circuitry22 may be used to allow data to be received by device 10 from externalequipment (e.g., a computer or other electrical equipment) and to allowa user to provide device 10 with user input. Input-output circuitry 22may also be used to gather information on the environment in whichdevice 10 is operating. Output components in circuitry 22 may allowdevice 10 to provide a user with output and may be used to communicatewith external electrical equipment.

As shown in FIG. 1, input-output circuitry 22 may include a camera suchas camera 40. Camera 40, which may sometimes be referred to as afront-facing camera, scene camera, or real-world-image camera, may beused in capturing images of the real-world (e.g., the user'senvironment). The real-world images captured by camera 40 may includecontent in the user's field of view and content that lies outside of theuser's field of view (e.g., portions of the user's environment that lieadjacent to the user's field of view).

Input-output circuitry 22 may also include a display such as display 14.Display 14 may be used to display images for a user of device 10.Display 14 may be an organic light-emitting diode display, a liquidcrystal display, a liquid-crystal-on-silicon display, a micromirrorarray display (e.g., a microelectromechanical systems (MEMS) display,sometimes referred to as a digital micromirror device), or any othersuitable display. Display 14 may be mounted in device 10, so that imageson display 14 are provided in the user's field of view. In someconfigurations, display 14 may be supported on a head-mounted supportstructure (glasses, helmet, hat, goggles, etc.). In otherconfigurations, display 14 may be mounted in a vehicle or otherequipment (e.g., in a location in which images may be presented to auser by reflecting image output from the display into the user's fieldof view from the interior surface of a vehicle window).

Display 14 may lie directly in the user's line of sight or an opticalcombiner in the user's line of sight may be used to merge images fromdisplay 14 with the user's view of the environment surrounding device10. In some arrangements, display 14 is housed in the housing of acellular telephone, tablet computer, or other portable electronicdevice. In general, display 14 may be placed in any suitable locationfor providing images to a user. During operation, a user may viewreal-world image content presented with display 14 (e.g., real-worldimages captured with a camera and displayed on display 14) and/or mayview real-world content directly (e.g., when the real world is visiblethrough an optical combiner that is being used in presenting images fromdisplay 14 for the user). Illustrative configurations in which usersview real-world content on display 14 that is overlaid with supplementalcontent on display 14 are sometimes described herein as an example.

Input-output circuitry 22 may include a gaze tracking system such asgaze tracking system 16. Gaze tracking (eye monitoring) system 16 mayinclude image sensors, light sources, and/or other equipment that isused in monitoring the eyes of the user. Gaze tracking system 16 mayinclude, for example, one or more visible and/or infrared cameras thatface a user's eyes and capture images of the user's eyes. Duringoperation of device 10, control circuitry 20 may use gaze trackingsystem 16 to track a user's gaze. Cameras and/or other sensors in system16 may, for example, determine the location of a user's eyes (e.g., thecenters of the user's pupils) and may determine the direction in whichthe user's eyes are oriented (the direction of the user's gaze,sometimes referred to as a measured point of gaze). By processing thisinformation and information on the location of display 14, the locationof the user's point of gaze may be determined dynamically. If desired,additional information (e.g., information from an inertial measurementunit containing an accelerometer, compass, and/or gyroscope and/orinformation from other motion sensors) may be used in determining thelocation of the user's point of gaze. For example, an inertialmeasurement unit, a visual odometry sensor in circuitry 22, and/or othersensors can determine the orientation of device 10 and/or a user's headrelative to the surrounding environment and can use this information inaddition to gaze tracking information to determine the current point ofgaze of the user.

User input and other information may also be gathered using sensors andother input devices in input-output devices 18. Input-output devices 18may include, for example, position and motion sensors (e.g., compasses,gyroscopes, accelerometers, and/or other devices for monitoring thelocation, orientation, and movement of device 10), may include forcesensors, temperature sensors, touch sensors, buttons, capacitiveproximity sensors, light-based proximity sensors, other proximitysensors, color ambient light sensors and other ambient light sensors,strain gauges, gas sensors, pressure sensors, moisture sensors, magneticsensors, gesture sensors, depth sensors (e.g., three-dimensionalstructured light sensors and other depth sensors), and other sensors,may include audio components such as microphones for gathering voicecommands and other audio input, and may include speakers for providingaudio output (e.g., for providing sound to the left and right ears of auser). If desired, input-output devices 18 may include haptic outputdevices (e.g., vibrating components), light-emitting diodes, lasers, andother light sources, and other output components.

Circuitry 22 may include wired and wireless communications circuitrythat allows device 10 (e.g., control circuitry 20) to communicate withexternal equipment (e.g., remote controls, joysticks and other inputcontrollers, portable electronic devices, computers, displays, etc.) andthat allows signals to be conveyed between components (circuitry) atdifferent locations in device 10. Global Positioning System (GPS)receiver circuitry and/or other satellite navigation system circuitry ininput-output circuitry 22 and/or other location sensor circuitry incircuitry 22 may be used in determining the user's location and/orvelocity. Using wireless communications circuitry in circuitry 22,device 10 can gather information over a network such as the internet(e.g., weather information, traffic conditions, etc.)

FIG. 2 is a diagram showing how gaze tracking system 16 may gather eyeposition information on a user's eye 30. In a typical scenario, gazetracking system 16 may include components for simultaneously monitoringtwo eyes such as eye 30 of FIG. 2.

As shown in FIG. 2, system 16 may include one or more image sensors suchas gaze tracking camera(s) 24. Each of cameras 24 may be focused on thefront of a user's eye such as eye 30 so that the characteristics of theuser's eye can be measured. One or more light-emitting diodes, lasers,lamps, and/or other light-emitting components may be used to form alight source for gaze tracking system 16 (see, e.g., light source 26).

During operation, light source 26 may emit light 28. For example, lightsource 26 may emit multiple beams of light 28 towards the user's eye 30(e.g., 2-10 beams of light, 6 beams of light, 4-7 beams of light, 4-9beams of light, etc.). Light 28 may be reflected off of the surface ofeye 30. Camera 24 may gather images of eye 30. Information on theappearance of eye 30 (e.g., iris information, pupil information, bloodvessel information, etc.) and/or information on reflected light (e.g.,one or more light beams) from cornea 32 and other portions of eye 30 maybe used by control circuitry 20 to determine the location of pupilcenter 36 of pupil 34 and the direction in which the user is currentlygazing (gaze direction 38). The eye position information (pupil centerinformation, eye orientation, etc.) that is gathered by gaze trackingsystem 16, the resulting information determined on the user's gazedirection 38, and information on the location of real-world objects(e.g., items in the environment surrounding device 10) relative tosystem 16 and eyes 30 may be used by control circuitry 20 to dynamicallyidentify the user's point of gaze within the real-world environmentsurrounding device 10. Real-world images can then be provided withaugmentations and/or enhanced based on the point of gaze (e.g., thelocation in the real world where the user's gaze lands).

During operation, control circuitry 20 can use camera 40 to captureimages of the real world and can use gaze tracking system 16 and otherinput-output devices 18 to gather user input and information on theuser's environment. Data on the user such as the user's glassesprescription and other information on the user (e.g., visual impairmentdetails such as low vision details) can also be gathered. Thisinformation can be used by control circuitry 20 in displaying mixedreality content for the user on display 14. For example, text in areal-world image can be magnified to accommodate a low vision user.Contrast and other image attributes can also be adjusted.

If desired, content attributes such as the identities of people in thereal-world image can be obtained by control circuitry 20 (e.g., byperforming pattern recognition operations). Mixed reality content basedon these content attributes can be presented to the user. For example,the names of people in the user's field of view may be presented to theuser by overlapping text boxes on a real world image. A text boxcontaining the name of each person in the user's field of view may beplaced adjacent to that person or only those people near to the user'spoint of gaze may be labeled. Point of gaze information may be used indetermining which people are being observed by the user.

Point of gaze information (e.g., information on the location and motionof the user's point of gaze) can also be analyzed to determine whensupplemental content should be presented based on real-world imagecontent outside of the user's field of view (or only in the user'sextreme peripheral vision). For example, if a user's point of gaze ismoving to the left, an icon can be presented for the user that indicatesthat content of interest (e.g., a moving vehicle) lies just to the leftof the user's field of view. In this way, a user may be provided withadvance notice of objects that are currently not being viewed by theuser.

Consider, as an example, real-world image 50 of FIG. 3. Real-world image50 may be captured by input-output circuitry 22 using camera 40. Ifdesired, the field of view of camera 40 may be larger than the user'sfield of view (see, e.g., user's field of view 52). Image 50 may containreal-world objects such as objects 58, 56, and 54 (e.g., people,vehicles, buildings, furniture, landscape elements, etc.). Duringoperation of device 10, mixed reality content may be presented to a userin some or all of the user's field of view. As an example,computer-generated images (still and/or moving image content with labelsand other text and/or graphics) and/or portions of real-world image 50that have been enhanced (e.g., by image processing to adjustmagnification, contrast, color, brightness, edge detail, etc.) can bepresented in one or more regions such as supplemental image region 60(sometimes referred to as a supplemental content region, overlay region,etc.).

The information that is presented in region 60 may, as an example,include a portion of real-world image 50 that has been enhanced by imageprocessing. The portion of real-world image 50 that is selected forimage enhancement may be selected based on user input and otherinformation. As an example, point-of-gaze information may be used indetermining which portion of real-world image 50 is to be magnified orotherwise enhance for display in region 60. If, for example, a user isviewing object 58 and the user's point of gaze overlaps object 58 (see,e.g., point-of-gaze 62), a portion of real-world image 50 that isassociated with some or all of object 58 may be enhanced (e.g.,enlarged, provided with additional contrast, etc.) and the resultingenhanced image can be displayed in region 60. If desired, some or all ofthe portion of the image covering object 58 may be enhanced when theuser's point of gaze does not overlap object 58, but instead lies withina predetermined distance of object 58. For example, control circuitry 20can display enhanced image content in response to detecting that theuser's point of gaze lies within area 66 adjacent to object 62 (see,e.g., point of gaze 64).

In determining when a user's point of gaze overlaps a desired object oran area near the desired object, control circuitry 20 can evaluate howlong the user's gaze dwells at different locations in the image. Athreshold point-of-gaze dwell time or other suitable criteria can beapplied to point-of-gaze information to determine whether the user'spoint of gaze has dwelled in a particular location sufficiently long totrigger the display of content in region 60. As an example, a magnifiedversion of object 62 may be presented in region 60 in response todetecting that the user's point of gaze has dwelled within region 66 formore than at least 0.2 s, at least 0.5 s, at least 1 s, at least 2 s, atleast 4 s, at least 10 s, less than 8 s, less than 5 s, or othersuitable time.

In some configurations, control circuitry 20 monitors movement in theuser's point of gaze and takes action based on point-of-gaze movement.If, in the example of FIG. 3, the user's point of gaze 68 is moving indirection 70 towards left edge 72 of the user's field of view 52 (e.g.,with a speed of movement that exceeds a predetermined threshold speedfor more than a predetermined threshold amount of time), controlcircuitry 20 can analyze image 50 for objects of interest that lieoutside of field of view 52 (e.g., items located beyond left edge 72such as object 54). Pattern recognition operations may be performed onimage 50 to determine which elements of image 50 correspond to objectsof interest. For example, image 50 may be processed by control circuitry20 to identify vehicles, people, objects moving more than a particularspeed, objects having a brightness that exceeds a predeterminedbrightness threshold, objects of a particular color, objects containingtext, etc. In response to determining that the user's point of gaze 68is moving in direction 70 and that an item of interest in image 50 suchas object 54 lies in direction 70 outside of the user's current field ofview 52, associated supplemental information can be presented for theuser. For example, control circuitry 20 can display a graphical elementsuch as arrow 74 that points toward out-of-view objects such as object54. This directs the user's attention to potentially important objectsthat are just beyond the user's field of view and helps the user bealert to such objects. If, as an example, a user is driving a vehicle oris walking along a street, it may be helpful to be informed that object54 (e.g., a fast-moving vehicle) is present. In some arrangements,disorientation and motion sickness can be avoided by suppressing thedisplay of magnified text or other supplemental content in response todetecting motion in the user's point of gaze (e.g., eye saccades).

Any suitable criteria may be used to determine when control circuitry 20should display supplemental information on real-world images (e.g.,overlaid content such as enhanced image content in region 60 and/or textor graphics such as arrow 74, a text label, a warning, etc.). As anexample, control circuitry 20 can determine the location of the user'spoint of gaze and the movement of the user's point of gaze by processingpoint of gaze location information from gaze tracking system 16 andinformation on the orientation of the user's head from a position andmotion sensor (orientation sensor) such as an inertial measurement unitand/or visual odometry system. If the location and/or motion of thepoint of gaze meets predetermined criteria (e.g., if the point of gazedwells in a particular location for more than a predetermined amount oftime, if the point of gaze moves at particular speeds and/ordirections), appropriate supplemental information can be displayed. Asanother example, supplemental information can be displayed based oninformation such as the location of device 10 (e.g., the location ofdevice 10 determined by a satellite navigation system receiver and/orother location sensor in input-output devices 18), sensor information(e.g., information on ambient lighting conditions indicating whether itis day or night), based on content in image 50 (e.g., whether faces arepresent and/or whether faces that are present correspond to particularindividuals, whether image 50 contains particular objects, ischaracterized by a predetermined amount of motion, has particularbrightness levels or contains particular colors or patterns, etc.),and/or based on other information. The display of supplementalinformation can also be suppressed (e.g., temporarily) whenever certaincriteria are satisfied (e.g., excessive motion in the user's point ofgaze, etc.).

If desired, a user of device 10 can press a button, speak a voicecommand, place the user's gaze direction in a particular orientation(e.g., gazing downward to invoke automatic bifocals, gazing towards theupper right, gazing so that the user's point of gaze overlaps adisplayed interactive icon, etc.), blink (e.g., three times in rapidsuccession so that camera 24 or other sensors can detect a predeterminedblink command), or can otherwise take an action that directs controlcircuitry 20 to display the supplemental information.

The conditions determining whether supplemental information is displayedand the type of supplemental information that is presented (e.g.,magnified image content, image content with enhanced contrast, etc.) caninclude user data such as the user's eyeglass prescription and othervisual function information. As an example, the user's vision can becharacterized using ophthalmic instruments (e.g., to characterize theuser's field of view, visual acuity, contrast sensitivity, andgeometrical distortions). This information can then be used indetermining whether, as an example, image content should be enhanced byincreasing contrast or by increasing magnification (or both). In somesituations (e.g., when the user's point of gaze is moving), motionsickness may be induced if magnified content is presented. Accordingly,the presentation of supplemental information such as enhanced imageinformation in region 60 may be suppressed when more than apredetermined amount of point of gaze motion is detected or othersuitable supplemental information suppression criteria are satisfied.

Supplemental information (e.g., computer-generated text or graphics orother augmentation information and/or enhanced image content) may bedisplayed in any suitable portion of the user's field of view. Asexamples, supplemental information may be displayed in region 60 of FIG.4 (e.g., in the middle of the displayed image), in region 60 of FIG. 5(e.g., along an edge of the displayed image such as along the bottomedge of the displayed image), in region 60 of FIG. 6 (e.g., in the upperright corner of the displayed image or other corner of the displayedimage), and/or in other suitable portion(s) of the image being displayedby display 14.

A flow chart of illustrative operations involved in using device 10 toprovide a user with real-world images and supplemental information ondisplay 14 is shown in FIG. 7.

During the operations of block 80, control circuitry 20 may useinput-output circuitry 22 to gather information on the user's gaze. Gazetracking system 16 and an inertial measurement unit and/or other sensorsin input-output devices 18 may be used to monitor eye movement andoptionally other input such as head movement/orientation and/orinformation on the orientation of device 10 and thereby measure theuser's point of gaze. Information on the user's point of gaze such aspoint of gaze location and/or point of gaze movement can be analyzed todetermine if a user is providing intentional input (gazing at apredetermined region of the displayed image for a predetermined time tointentionally invoke the display of supplemental information) and/or canbe analyzed to determine whether the user is viewing a particular itemof interest or is moving toward a particular item of interest (asexamples). The operations of block 80 may include characterizing theuser's eye motion as being fixated (not moving), saccade (moving), andblink.

During the operations of block 82, control circuitry 20 can analyzecaptured images such as image 50. For example, images of the scene theuser is viewing (e.g., real world image 50) can be processed using imagerecognition algorithms and other processing algorithms to identifycontent in the image and other attributes. As an example, the image canbe processed to identify image attributes such as brightness, colorbreakdown, contrast, etc. Patterns can be detected (e.g., to identifyvehicles, roads, people, faces, the identities of people associated withparticular faces, road signs with and without text, text on billboardsand other advertising, and/or other items in the user's environment).Object edges can be detected using edge detection processes. Objects atrest and objects in motion can be categorized. All information in image50 can be processed using techniques such as these or, if desired,subsets of the information in image 50 can be processed. As an example,image processing such as facial recognition processing can be performedfor those faces that a user's point of gaze dwells on, but can beskipped for other faces to reduce processing load. The processing ofthis information can also be influenced by other sensor output. Forexample, road signs information can be processed and identified formagnification only when the user and device 10 are moving faster than apredetermined speed (indicating that the user is driving).

During the operations of block 84, information on the vision and otherattributes of the user can be gathered. For example, user data such asvisual impairment information can be gathered using ophthalmologicinstruments and provided to control circuitry 20 (e.g., over a wiredand/or wireless communications path, using user input devices, etc.). Bycharacterizing the user's acuity, contrast sensitivity, field of view,geometrical distortions and other vision attributes, appropriatelytargeted image enhancements and augmentations can be provided for theuser.

During the operations of block 86, user input can be gathered usinginput-output circuitry 22. For example, gaze tracking system 16 maygather intentional eye input such as eye blinks and intentionalpoint-of-gaze input (e.g., eye gestures such as gestures involvingmovement of the point of gaze of the user in and out of a particularregion or intentional direction of the user's gaze downward toautomatically invoke a magnified strip of image 50 serving as digitalbifocals, etc.). Circuitry 22 may also use buttons, touch sensors,microphones, and other input-output circuitry to gather user input. Userinput may be used to place device 10 in different operations modes(e.g., walking mode in which supplemental information is provided in thecenter of a displayed image, a driving mode in which supplementalinformation is provided along an edge of the display image, a meetingmode in which facial recognition operations are used to identify andlabel participates of a meeting, an inactive mode in which nosupplemental information is displayed, etc.). User input may also beused to supply device 10 with preferences (e.g., user adjustablesettings such as preferred display brightness level, threshold values,magnification settings, a desired location for supplemental informationregion 60, etc.).

Additional input may be gathered with input-output circuitry 22 duringthe operations of block 88. For example, control circuitry 20 may usecircuitry 22 to gather location information, information on the speedand direction of motion of device 10, information on traffic conditions,information on local weather, etc.

During the operations of block 90, control circuitry 20 may display thecaptured image from camera 40 on display 14 for viewing by the user(and/or the user can view the real world through an optical combinersuch as a vehicle window or an optical combiner in a head-mounteddevice). Supplemental information can be displayed on display 14 so asto overlap the real world content. For example, information can bedisplayed over some or all of display 14 that includes augmented visioninformation such as graphics, text, and other computer-generatedoverlays and/or that includes enhanced vision information such as all orsome of image 50 that has been processed to adjust contrast, to adjustbrightness, to adjust color, to adjust object features such as toenhance edges by applying edge enhancement, to adjust magnification, toadjust geometric distortion, or to adjust other image attributes. Userdata gathered during the operations of block 84 (e.g., a user's visualimpairment details) may be taken into account during the operations ofblock 90. For example, if the user's vision is characterized by ageometric distortion, a compensating geometric distortion can be appliedin displaying the image on display 14. If the user has low visualspatial acuity, content can be magnified appropriately.

In general, supplemental information can be displayed based on theinformation gathered during block 80 (e.g., point of gaze information),image analysis performed during block 82, information gathered duringblock 84, information gathered during block 86 (e.g., mode selectioninformation and other information supplied as user input), and/orinformation gathered during the operations of block 88 (e.g.,information on the current weather, traffic conditions, etc.). Imageadjustments can be made locally (e.g., to magnify text in an appropriateregion such as regions 60 of FIGS. 4, 5, and 6) or may be appliedglobally (e.g., by enhancing contrast for all of the displayed imagewhen it is determined that the user's vision is characterized by lowcontrast).

The foregoing is merely illustrative and various modifications can bemade to the described embodiments. The foregoing embodiments may beimplemented individually or in any combination.

What is claimed is:
 1. An electronic device, comprising: a cameraconfigured to capture an image of the real world; a display configuredto display the image; a gaze tracking system configured to gatherpoint-of-gaze information; and control circuitry that is configured todisplay magnified supplemental content in the image on the display basedon the point-of-gaze information.
 2. The electronic device defined inclaim 1 wherein the magnified supplemental content comprises a magnifiedportion of the image of the real world.
 3. The electronic device definedin claim 2 wherein the control circuitry is configured to display themagnified portion of the image of the real world as an overlay on aregion of the image of the real world.
 4. The electronic device definedin claim 3 wherein the region is characterized by a location within theimage of the real world and wherein the electronic device comprisesinput-output circuitry configured to gather user input to adjust thelocation.
 5. The electronic device defined in claim 2 wherein thecontrol circuitry is configured to display the magnified portion inresponse to a point-of-gaze gesture.
 6. The electronic device defined inclaim 1 wherein the control circuitry is configured to display themagnified supplemental content in response to user eye blink inputgathered with the gaze tracking system.
 7. The electronic device definedin claim 1 wherein the control circuitry is configured to display themagnified supplemental content in response to determining from the pointof gaze information that a point of gaze has dwelled on a portion of theimage for more than a predetermined point-of-gaze dwell time.
 8. Theelectronic device defined in claim 1 wherein the control circuitry isconfigured to display an augmentation including computer-generatedgraphic content in the image on the display based on the point-of-gazeinformation.
 9. The electronic device defined in claim 1 wherein thecontrol circuitry is configured to display the magnified supplementalcontent based on information on a user's vision attributes.
 10. Theelectronic device defined in claim 1 wherein the control circuitry isconfigured to analyze the image to identify objects in motion.
 11. Theelectronic device defined in claim 1 wherein the control circuitry isconfigured to analyze the image to identify faces and wherein thecontrol circuitry is configured to display text that labels anidentified face with an identity associated with that identified face.12. The electronic device defined in claim 1 wherein the controlcircuitry is configured to analyze the image to identify text andwherein the magnified supplemental content includes a magnified versionof the text.
 13. The electronic device defined in claim 1 furthercomprising input-output circuitry, wherein the control circuitry isconfigured to gather user input with the input-output circuitry andwherein the control circuitry is configured to display the magnifiedsupplemental content based on an operating mode selected with the userinput.
 14. The electronic device defined in claim 13 wherein theinput-output circuitry comprises location sensing circuitry and whereinthe control circuitry is configured to display the magnifiedsupplemental content based on a geographic location identified with thelocation sensing circuitry.
 15. The electronic device defined in claim 1wherein the control circuitry is configured to provide the magnifiedsupplemental information based on a point-of-gaze dwell time associatedwith the point of gaze information, attributes in the image, user visionattributes, and user input.
 16. The electronic device defined in claim 1further comprising a motion sensor, wherein the control circuitry isconfigured to display the magnified supplemental content in response todetermining that a speed measured by the motion sensor exceeds apredetermined limit.
 17. An electronic device, comprising: a cameraconfigured to capture an image of the real world; a display; a gazetracking system configured to gather point of gaze information; andcontrol circuitry that is configured to display enhanced image contentthat overlaps the image on the display based on the point of gazeinformation.
 18. The electronic device defined in claim 17 wherein thecontrol circuitry is configured to gather user vision informationcomprising a user vision characteristic selected from the groupconsisting of: visual acuity, contrast, field of view and geometricaldistortions and is configured to display the enhanced image contentbased on the gathered user vision information.
 19. The electronic devicedefined in claim 17 wherein the enhanced image content comprises anenhanced portion of the image selected from the group consisting of: amagnified portion of the image; a portion of the image with increasedcontrast, a portion of the image with enhanced edges, a portion of theimage with altered color.
 20. An electronic device, comprising: a cameraconfigured to capture an image of the real world; gaze trackingcircuitry configured to gather point of gaze information; a display; andcontrol circuitry configured to: analyze the image of the real world toidentify text in the image; and display a magnified version of the textoverlaid on part of the image on the display based on the point of gazeinformation.